Compsci 650 Applied Information Theory Lecture 4

Since the appearing probability of each English symbol P (a), ...P (b), ...P (” ”), ..., P (; ) is not uniform, we should be able to reduce the number of required bits. Based on the probability of each English symbol, we can compute the entropy H(E) ∼= 4.5 bits / char. If we use Huffman coding taught in this lecture to encode English keyboard, then we only need around 4.7 bits / char. Furthermore, the characters in English are not independent. For example, the number of meaningful English strings with length 8 should be much smaller than 96 and some strings occur much often than the others. In fact, we can get the entropy of such string H(E) ∼= 2.4 bits/char. If we further extend the length of string to infinity, the estimated entropy of English would be H(E∞) ∼= 1.3 bits / char. Using lossless data compression algorithms to encode English keyboard results in the following: LZW ∼= 3.7 bits / char GZip ∼= 2.7 bits / char BW ∼= 1.89 bits / char where LZW will be taught in the future lecture, and BW is an industrial standard, which achieves good performance by combining LZW, Gzip and several other compression algorithms. Nowadays, the cost of storage becomes much cheaper, but the internet bandwidth still remains an expensive resource. This is one of reasons why the compression techniques still play important roles in our life.